{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final Remarks\n",
    "\n",
    "In this tutorial, we took an approach where the same algorithm was ported to GPU using different popular methods. Each method has its strengths and suffices the purpose for which it was created. From a developer's point of view below listed are some key parameters which are crucial to any development exercise:\n",
    "\n",
    "- **Ease of Programming**: How much in-depth knowledge of processor architecture is required for a developer before converting the code to GPU?\n",
    "- **Performance**: How much effort is required to reach desirable performance on a particular architecture.\n",
    "- **Portability**: To what extent does the same code run on multiple architectures? What provisions are provided by programming approach to target different platforms?\n",
    "- **Support**: The overall ecosystem and support by the community.\n",
    "    - Which all compilers implement the standard?\n",
    "    - Which all languages are supported?\n",
    "    - Which all applications make use it?\n",
    "    - How easy or difficult it is to profile/debug the application.\n",
    "    \n",
    "Let us try to create a high-level buckets for each of this parameter above with a limited scope of GPU support:\n",
    "\n",
    "| | |  |  | \n",
    "| :--- | :--- | :--- | :--- |\n",
    "| Ease of Programming | Low: Minimal architecture specific knowledge needed  | Intermediate: Mimimal changes expected in code design.  Using these along with architecture knowledge helps in better performance | High: In-Depth GPU architecture knowledge must |\n",
    "| Performance  | Depends: Based on the complexity/type of application the performance may vary | High: Exposes methods to get good performance. These methods are integral part of design and exposed to programmer at various granularities | Best: Full control to developers to control parallelism and memory access |\n",
    "| Portability | Integral: Part of the key objective  | Limited: Works only on specific platform | | \n",
    "| Support | Established: Proven over years and support by multiple vendors for GPU | Emerging: Gaining traction by multiple vendors for GPU  | |\n",
    "\n",
    "There is a very thin line between these categories, and within that limited scope and view we could categorize different approaches as follows:\n",
    "\n",
    " \n",
    "| | OpenACC | OpenMP | stdpar and DO CONCURRENT| Kokkos | CUDA Laguages |\n",
    "| --- | --- | --- | --- | --- | --- |\n",
    "| Ease | High  | High | High  | Intermediate | Low |\n",
    "| Performance  | Depends | Depends | Depends | High | Best |\n",
    "| Portability | Integral  | Integral | Integral | Integral | Limited |\n",
    "| Support | Established | Emerging | Emerging | Established | Established |\n",
    "\n",
    "<!--\n",
    "| | OpenACC | OpenMP | DO-CONCURRENT | Kokkos | CUDA Laguages |\n",
    "| --- | --- | --- | --- | --- | --- |\n",
    "| Ease | High  | High | High  | Intermediate | Low |\n",
    "| Performance  | Depends | Depends | Depends | High | Best |\n",
    "| Portability | Integral  | Integral | Integral | Integral | Limited |\n",
    "| Support | Established | Emerging | Emerging | Established | Established |\n",
    "-->\n",
    "\n",
    "Below given are points that will help users as no one programming model fits all needs.\n",
    "\n",
    "## Ease of Programming\n",
    "- The directive‐based OpenMP and OpenACC programming models are generally least intrusive when applied to the loops. \n",
    "<!--- Kokkos required restructuring of the existing code for the parallel dispatch via functors or lambda functions-->\n",
    "- CUDA required a comparable amount of rewriting effort, in particular, to map the loops onto a CUDA grid of threads and thread blocks\n",
    "- stdpar also required us to change the constructs to make use of C++17 templates. It may be preferred for new developments having C++ template style coding. \n",
    "- DO-CONCURRENT also required us to make minimal changes by replacing the *do* loop to *do concurrent* . \n",
    "- The overhead for OpenMP and OpenACC in terms of lines of code is the smallest, followed by stdpar and Kokkos\n",
    "\n",
    "## Performance\n",
    "While we have not gone into the details of optimization for any of these programming models the analysis provided here is based on the general design of the programming model itself.\n",
    "\n",
    "<!--- Kokkos when compiled enables the use of correct compiler optimization flags for the respective platform, while for the other frameworks, the user has to set these flags manually. This gives kokkos an upper hand over OpenACC and OpenMP. -->\n",
    "- OpenACC and OpenMP abstract model defines a least common denominator for accelerator devices, but cannot represent the architectural specifics of these devices without making the language less portable.\n",
    "- stdpar and DO-CONCURRENT, on the other hand, are more abstract and gives less control to developers to optimize the code\n",
    "\n",
    "## Portability\n",
    "<!--We observed the same code being run on both multicore and GPU using OpenMP, OpenACC, Kokkos and stdpar. -->\n",
    "We observed the same code being run on both multicore and GPU using OpenMP, OpenACC, stdpar and DO-CONCURRENT. The point we highlight here is how a programming model supports the divergent cases where developers may choose to use the different directive variants to get more performance. In a real application, the tolerance for this portability/performance trade-off will vary according to the needs of the programmer and application.\n",
    "- OpenMP supports [Metadirective](https://www.openmp.org/spec-html/5.0/openmpsu28.html) where the developer can choose to activate different directive variants based on the condition selected.\n",
    "- In OpenACC when using ```kernel``` construct, the compiler is responsible for mapping and partitioning the program to the underlying hardware. Since the compiler will mostly take care of the parallelization issues, the descriptive approach may generate performance code for s specific architecture. The downside is that the quality of the generated accelerated code depends significantly on the capability of the compiler and hence the term \"may\".\n",
    "## Support\n",
    "<!--- Kokkos project is very well documented and the developers support on GitHub is excellent -->\n",
    "- OpenACC implementation is present in most popular compilers like NVIDIA HPC SDK, PGI, GCC, Clang and CRAY.\n",
    "- OpenMP GPU support is currently available on limited compilers, but being the most supported programming model for multicore, it is a matter of time when it comes at par with other models for GPU support.\n",
    "- stdpar being part of the C++ standard, is bound to become an integral part of most compilers supporting parallelism.\n",
    "- DO-CONCURRENT being part of the ISO Fortran standard, is bound to become an integral part of most compilers supporting parallelism.\n",
    "\n",
    "\n",
    "Parallel Computing, in general, has been a difficult task and requires developers not just to know a programming approach but also to think in parallel. While this tutorial provides you with a good start, it is highly recommended to go through Profiling and Optimization bootcamps as the next steps.\n",
    "\n",
    "-----\n",
    "<!--\n",
    "# <div style=\"text-align: center ;border:3px; border-style:solid; border-color:#FF0000  ; padding: 1em\">[HOME](../../nways_MD_start.ipynb)</div>\n",
    "-->\n",
    "-----\n",
    "\n",
    "# Links and Resources\n",
    "[OpenACC API guide](https://www.openacc.org/sites/default/files/inline-files/OpenACC%20API%202.6%20Reference%20Guide.pdf)\n",
    "\n",
    "[NVIDIA Nsight System](https://docs.nvidia.com/nsight-systems/)\n",
    "\n",
    "[NVIDIA Nsight Compute](https://developer.nvidia.com/nsight-compute)\n",
    "\n",
    "[CUDA Toolkit Download](https://developer.nvidia.com/cuda-downloads)\n",
    "\n",
    "**NOTE**: To be able to see the Nsight Systems profiler output, please download Nsight Systems latest version from [here](https://developer.nvidia.com/nsight-systems).\n",
    "\n",
    "Don't forget to check out additional [Open Hackathons Resources](https://www.openhackathons.org/s/technical-resources) and join our [OpenACC and Hackathons Slack Channel](https://www.openacc.org/community#slack) to share your experience and get more help from the community.\n",
    "\n",
    "--- \n",
    "\n",
    "### Other Bootcamps\n",
    "The contents of this Bootcamp originates from [OpenACC GPU Bootcamp Github](https://github.com/gpuhackathons-org/gpubootcamp).  Here are some additional Bootcamp which might of interest: \n",
    "\n",
    "- [AI for HPC](https://github.com/gpuhackathons-org/gpubootcamp/tree/master/hpc_ai/ai_science_climate)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Licensing \n",
    "\n",
    "Copyright © 2022 OpenACC-Standard.org.  This material is released by OpenACC-Standard.org, in collaboration with NVIDIA Corporation, under the Creative Commons Attribution 4.0 International (CC BY 4.0). These materials may include references to hardware and software developed by other entities; all applicable licensing and copyrights apply."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
